User interface method

ABSTRACT

A computer-implemented method includes detecting, at a wearable computing device, a first direction of a first stare, wherein the wearable computing device includes a head-mountable display unit, identifying a target based on the detected first direction, and based on a determination that a first time duration of the first stare is greater than or equal to a first predetermined time threshold, identifying information relevant to the target and displaying the identified information on the display unit. Subsequent to displaying the identified information, the method includes detecting a second stare that is directed at the target or at the displayed information, and based on a determination that a second time duration of the second stare is greater than or equal to a second predetermined time threshold, identifying additional information relevant to the target, and displaying the additional information on the display unit.

BACKGROUND

Unless otherwise indicated herein, the materials described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

A head mounted display (HMD) unit is a display device, worn on a user'shead, which can display information via images in front of one eye(monocular HMD unit) or each eye (binocular HMD unit) of the user. HMDunits have been used for a wide range of applications.

HMD units, also referred to as wearable systems, can integrate variouselements, such as miniaturized computers, input devices, sensors,detectors, image displays, wireless communication devices as well asimage and audio processors, into a device that can be worn by a user.Such devices provide a mobile and lightweight solution to communicating,computing and interacting with one's environment. With the advance oftechnologies associated with wearable systems and miniaturized opticalelements, it has become possible to consider wearable compact opticaldisplays that augment the wearer's experience of the real world.

By placing an image display unit close to the user's eye(s), anartificial image can be made to overlay the wearer's view of the realworld. The wearable systems incorporating these image display units alsoreferred to as “near-eye see-through displays”, or “heads-up displays”(HUDs). Depending upon the size of the display unit and the distance tothe wearer's eye, the artificial image may fill or nearly fill theuser's field of view.

SUMMARY

In one embodiment, a computer-implemented method includes detecting, ata wearable computing device, a first direction of a first stare, whereinthe wearable computing device includes a head-mountable display unit,identifying a target based on the detected first direction, and based ona determination that a first time duration of the first stare is greaterthan or equal to a first predetermined time threshold, identifyinginformation relevant to the target and displaying the identifiedinformation on the display unit. Subsequent to displaying the identifiedinformation, the method includes detecting a second stare that isdirected at the target or at the displayed information, and based on adetermination that a second time duration of the second stare is greaterthan or equal to a second predetermined time threshold, identifyingadditional information relevant to the target, and displaying theadditional information on the display unit.

In another embodiment, a computer-implemented method includes detecting,at a wearable computing device, a direction of a stare, wherein thewearable computing device includes a head-mountable display unit,identifying a target based on the detected direction, and based on adetermination that a first time duration of the stare is greater than orequal to a first predetermined time threshold, identifying informationrelevant to the target and displaying the identified information on thedisplay unit. Subsequent to displaying the identified information, themethod further includes determining that a second time duration of thestare is greater than or equal to a second predetermined time threshold,and based on a determination that the second time duration of the stareis greater than or equal to the second predetermined time threshold,identifying additional information relevant to the target, anddisplaying the additional information on the display unit.

In another embodiment, a head-wearable display device includes an eyetracking unit for determining a stare behavior, a display unit fordisplaying graphical images, an image projecting unit for projecting thegraphical images on the display unit, a memory unit storing programinstructions for producing changes in information density and type onthe display unit, and a processor operatively coupled to the eyetracking unit, the memory, and the image projecting unit, and configuredto execute the program instructions.

In another embodiment, a non-transitory computer readable storage mediumcontains instructions that cause a computing system to perform themethod for producing changes in information density and type on thedisplay unit.

These as well as other aspects, advantages, and alternatives will becomeapparent to those of ordinary skill in the art by reading the followingdetailed description, with reference where appropriate to theaccompanying drawings. Further, it should be understood that thedisclosure provided in this summary section and elsewhere in thisdocument is intended to discuss the invention by way of example only andnot by way of limitation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a wearable computing device, inaccordance with an example embodiment;

FIG. 2A is a front view of a head-mounted display (HMD), in accordancewith an example embodiment;

FIG. 2B is a top view of the head-mounted display (HMD) of FIG. 2A, inaccordance with an example embodiment;

FIG. 2C is a side view of the head-mounted display (HMD) of FIG. 2A andFIG. 2B, in accordance with an example embodiment;

FIG. 3 is a flowchart of a method, in accordance with an exampleembodiment; and

FIGS. 4A-4C collectively depict changes in information density and typeon an HMD, in accordance with an example embodiment;

FIGS. 5A-5B collectively depict changes in information density and typeon an HMD, in accordance with an example embodiment; and

FIG. 6 is a schematic diagram illustrating a conceptual partial view ofan example computer program associated with the method of FIG. 3.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying figures, which form a part thereof. In the figures, similarsymbols typically identify similar components, unless context dictatesotherwise. The illustrative embodiments described in the detaileddescription and figures are not meant to be limiting. Other embodimentsmay be utilized, and other changes may be made, without departing fromthe spirit or scope of the subject matter presented herein. It will bereadily understood that the aspects of the present disclosure, asgenerally described herein, and illustrated in the figures, can bearranged, substituted, combined, separated, and designed in a widevariety of different configurations, all of which are contemplatedherein.

1. Overview

In one embodiment, a computing system is configured to image at leastone eye of a wearer of a head mounted display or device (“HMD”) in orderto determine a stare direction within a field of view of the HMD. TheHMD may be any suitable device that is capable of being worn or mountedon a user's or wearer's head. That is, to determine stare directions oneor both eyes of the wearer of the HMD could be imaged. From thedetermined stare direction, stare locations may be determined that couldcorrespond to locations where the HMD wearer may be looking within thedisplay area or in the real world. The wearer's stare may be directed ata target that figures on a display presented on the HMD. Alternatively,the wearer's stare may be directed at a target that is located withinthe real world seen through a see-through display unit of the HMD.

Exemplary methods and systems may be configured upon detection of auser's stare at a target for a predetermined time to display informationrelated to the target. Further, following the display of the informationand upon determination that the user continues staring at the target forat least another predetermined time, additional information related tothe target is displayed. As such, a computing system, including an eyetracking system, is configured to provide a progressive display ofadditional information about the target as the user keeps staring as thetarget, i.e., the longer the user stares the more information about thetarget is displayed. Alternatively, upon detection that the user hasbeen staring at the displayed target information, rather than at thetarget, for at least another predetermined time, the computing systemprovides additional information related to the target. The informationrelated to the target may be retrieved from a database integral to theHMD and/or from a remote server. The additional information may build onor may be related to the initial information in order to provide furtherdetails about a certain feature or features of the target.

In one embodiment, the HMD wearer's stares, deliberate or unconscious,may include: a dwelling of the wearer's eyes over a small area ofinterest, prolonging reading movements over a certain area, squinting,changes in pupil dilations, changes in eye lens accommodation, repeatedlooks at an area of interest, or other behaviors detectable by theeye-tracking system. Once the eye tracking system has detected a longstaring behavior at the target, a system processor may be configured tovary or modify the display in a variety of ways, such as for example:

-   -   Burning away a layer of information to reveal more details.    -   X-raying a top layer of information to reveal settings or other        options relevant to the top layer of information.    -   Providing additional content relevant to the stared-at target,        for example, fading in time of an e-mail when the eye-tracking        system detects that the wearer has been reading the subject line        repeatedly.

Certain illustrative examples of a system and method for producingchanges in information density and type in a display area of a HMD basedon eye stare information are described below. It is to be understood,however, that other embodiments are possible and are implicitlyconsidered within the context of the following example embodiments.

2. Head-Mounted Display Apparatus with Eye Tracking for Interaction witha User Interface

The HMD may enable the user or wearer to observe the wearer's real-worldsurroundings and also view a displayed image, such as acomputer-generated image or virtual image. In some cases, the displayedimage may overlay a portion of the wearer's field of view of the realworld. Thus, while the wearer of the HMD is going about his or her dailyactivities, such as walking, driving, exercising, etc., the wearer maybe able to see a displayed image generated by the HMD at the same timethat the wearer is looking out at his or her real-world surroundings.

The displayed image might include, for example, graphics, text, and/orvideo. The content of the displayed image may relate to any number ofcontexts, including but not limited to the wearer's current environment,an activity in which the wearer is currently engaged, the biometricstatus of the wearer, and any audio, video, or textual communicationsthat have been directed to the wearer. The images displayed by the HMDmay also be part of an interactive user interface. For example, the HMDcould be part of a wearable computing device. Thus, the images displayedby the HMD could include menus, selection boxes, navigation icons, orother user interface features that enable the wearer to invoke functionsof the wearable computing device or otherwise interact with the wearablecomputing device.

The images displayed by the HMD could appear anywhere in the wearer'sfield of view. For example, the displayed image might occur at or nearthe center of the wearer's field of view, or the displayed image mightbe confined to the top, bottom, or a corner of the wearer's field ofview. Alternatively, the displayed image might be at the periphery of orentirely outside of the wearer's normal field of view. For example, thedisplayed image might be positioned such that it is not visible when thewearer looks straight ahead but is visible when the wearer looks in aspecific direction, such as up, down, or to one side. In addition, thedisplayed image might overlay only a small portion of the wearer's fieldof view, or the displayed image might fill most or all of the wearer'sfield of view.

The virtual images could be displayed based on the position andorientation of the HMD. For example, the HMD may include position andorientation sensors so that when the user moves his or her head, dataregarding the position and orientation of the HMD can be received by theprocessor. The HMD may additionally include a display controllable bythe processor, so when the user moves his or her head, the processor mayadjust the displayed image on the display. In particular, the displayedimage may move in the opposite direction of head movement to create thesensation of looking around a world with superimposed virtual images.

Alternatively, in case the HMD moves relatively to the eyes of thewearer, the processor is configured to adjust a location, a size, and afocus of the displayed image so as to stabilize the displayed imagerelative to the eyes of the wearer.

As stated above and known to one of ordinary skill in the art, a stareor staredown is a fixed and intent look, and may be determined upondetection of a lengthy visual fixation or of a number of consecutivefixations at a target. As a difference to just looking at a display inan observational mode, the staredown involves looking at the display inan intentional mode. As such, the eye tracking system can be configuredto detect and measure the following characteristics of the staredown:location, duration, and repeated fixations.

By incorporating an eye-tracking system into the HMD, stares orstaredowns at the displayed image or through the display unit may bedetermined. That is, the eye-tracking system in combination with the HMDmay determine where the wearer of the HMD is looking at, detectstaredown behaviors, and dynamically adjust information or provideadditional information presented on the display unit as a result basedon the length of the stare. Staredown behaviors may also be unconsciousactions that accompany increased attention.

In one embodiment, staring down intently at a target found in thedisplayed image for a predetermined period of time may result in thattarget being selected by the system, and information related/relevant tothe target maybe searched and displayed on the display unit of the HMD.Moreover, when the system determines that the user's stare has lastedmore than the predetermined period of time, then additional informationrelevant to the target may be displayed. In another embodiment, thetarget being stared at may be seen through the display unit and locatedin a certain area of the visible real-world surroundings. Thus, thelength of the detected stares may act as input instructions that cancause the processor to increase information displayed.

FIG. 1 is a schematic diagram of a wearable computing device or ahead-mounted display (HMD) 100 that may include several differentcomponents and subsystems. As shown, the HMD 100 includes a computingsystem 101, which may include an eye-tracking system 102, a HMD-trackingsystem 104, an optical system 106, peripherals 108, a power supply 110,a processor 112, a memory 114, and a user interface 115. Eye-trackingsystem 102 may include hardware such as an infrared camera 116 and atleast one infrared light source 118. HMD-tracking system 104 may includea gyroscope 120, a global positioning system (GPS) 122, and anaccelerometer 124. Optical system 106 may include, in one embodiment, adisplay unit or panel 126, a display light source 128, and optics 130.Peripherals 108 may include a wireless communication interface 134, atouchpad 136, a microphone 138, a camera 140, and a speaker 142.

In an example embodiment, HMD 100 includes a see-through display. Thus,the wearer of HMD 100 may observe a portion of the real-worldenvironment, i.e., in a particular field of view provided by opticalsystem 106. In the example embodiment, HMD 100 is operable to displayvirtual images that are superimposed on the field of view, for example,to provide an “augmented reality” experience. Some of the virtual imagesdisplayed by HMD 100 may be superimposed over particular objects in thefield of view. HMD 100 may also display images that appear to hoverwithin the field of view instead of being associated with particularobjects in the field of view. Alternatively, virtual images could bedisplayed independently from the real-world environment.

Components of computing system 101 may be configured to work in aninterconnected fashion with other components within or outside theirrespective systems. For instance, in an example embodiment, infraredcamera 116 may image one or both of the HMD wearer's eyes. Infraredcamera 116 may deliver image information to processor 112, which mayaccess memory 114 and make a determination regarding the direction ofthe HMD wearer's stare, also termed a stare axis. Processor 112 mayfurther accept input from GPS unit 122, gyroscope 120, and/oraccelerometer 124 to determine the location and orientation of HMD 100.

HMD 100 could be configured as, for example, eyeglasses, goggles, ahelmet, a hat, a visor, a headband, or in some other form that can besupported on or from the wearer's head. Further, HMD 100 may beconfigured to display images to both of the wearer's eyes, for example,using two see-through displays. Alternatively, HMD 100 may include onlya single see-through display and may display images to only one of thewearer's eyes, either the left eye or the right eye. HMD 100 may alsorepresent an opaque display configured to display images to one or bothof the wearer's eyes without a view of the real-world environment.Further, HMD 100 could provide an opaque display for a first eye of thewearer as well as provide a view of the real-world environment for asecond eye of the wearer.

A power supply 110 may provide power to various HMD components and couldrepresent, for example, a rechargeable lithium-ion battery. Variousother power supply materials and types known in the art are possible.

The functioning of computing system 101 may be controlled by processor112 that executes instructions stored in a non-transitory computerreadable medium, such as memory 114. Thus, processor 112 in combinationwith instructions stored in memory 114 may function as a controller ofcomputing system 101. As such, processor 112 may control user interface115 to adjust the images displayed by HMD 100. Processor 112 may alsocontrol wireless communication interface 134 and various othercomponents of computing system 101. Processor 112 may additionallyrepresent a plurality of computing devices that may serve to controlindividual components or subsystems of the HMD 100 in a distributedfashion.

In addition to instructions that may be executed by processor 112,memory 114 may store data that may include a set of calibrated wearereye pupil positions and a collection of past eye pupil positions. Thus,memory 114 may function as a database of information related to staredirection. Such information may be used by computing system 101 toanticipate where the user will look and determine what images are to bedisplayed to the wearer. Calibrated wearer eye pupil positions mayinclude, for instance, information regarding the extents or range of thewearer's eye pupil movement (right/left and upwards/downwards) as wellas wearer eye pupil positions that may relate to various reference axes.

Reference axes could represent, for example, an axis extending from aviewing location, such as one eye of the user, and through a targetobject or the apparent center of a field of view (i.e. a central axisthat may project through a center point of the apparent display panel ofHMD 100). Other possibilities for reference axes may exist. Thus, areference axis may further represent a basis for determining a staredown direction.

In addition, information may be stored in memory 114 regarding possiblecontrol instructions that may be enacted via deliberate or unconsciouseye movements, described above. Control instructions could be based ondwell-based selection of a target. For instance, if a wearer fixatesvisually upon a particular virtual image or real-world object for longerthan a predetermined time period, a control instruction may be generatedto select the virtual image or real-world object as a target object.Many other control instructions are possible.

In an example embodiment, user interface 115 is configured to provideinformation to the wearer or receiving input from the wearer. Userinterface 115 could be associated with, for example, the displayedvirtual images and/or one or more input devices in peripherals 108, suchas touchpad 136 or microphone 138. Processor 112 may control thefunctioning of computing system 101 based on inputs received throughuser interface 115. For example, processor 112 may utilize user inputfrom user interface 115 to control how computing system 101 displaysimages within a field of view or to determine what images computingsystem 101 displays.

In an example embodiment, eye-tracking system 102 may deliverinformation to processor 112 regarding the eye position of a wearer ofHMD 100. The eye-tracking data can be used, for instance, to determine adirection in which the HMD wearer may be staring. Processor 112 maydetermine targets among the displayed images or from the real-world seenthrough the display unit based on information from eye-tracking system102. Processor 112 may control user interface 115 and display panel 126to produce and display new information related to a target, which issubject of a stare by the wearer, on the original display that includesthe target. Alternatively, the new information may be displayed on lens204 while the target is displayed on lens 202. Further, originaldisplays and new information displays may be presented to the wearer invarious ways.

Infrared camera 116 may be utilized by the eye-tracking system 102 tocapture images of a viewing location associated with HMD 100. Thus,infrared camera 116 may image the eye of a HMD wearer that may belocated at the viewing location. The images could be either video imagesor still images. The images obtained by infrared camera 116 regardingthe HMD wearer's eye may help determine where the wearer is looking,i.e., a viewing direction, within the HMD field of view, for instance byallowing processor 112 to ascertain the location of the HMD wearer's eyepupil. Analysis of the images obtained by infrared camera 116 could beperformed by processor 112 in conjunction with memory 114.

The imaging of the viewing location could occur continuously or atdiscrete times depending upon, for instance, user interactions with userinterface 115. Infrared camera 116 could be integrated into opticalsystem 106 or mounted on HMD 100. Alternatively, infrared camera 116could be positioned apart from HMD 100 altogether. Furthermore, infraredcamera 116 could additionally represent a conventional visible lightcamera with sensing capabilities in the infrared wavelengths.

Infrared light source 118 could represent one or more infraredlight-emitting diodes (LEDs) or infrared laser diodes that mayilluminate a viewing location. One or both eyes of a wearer of HMD 100may be illuminated by infrared light source 118. Infrared light source118 may be positioned along an optical axis common to infrared camera116, and/or infrared light source 118 may be positioned elsewhere.Infrared light source 118 may illuminate the viewing locationcontinuously or may be turned on at discrete times. Additionally, whenilluminated, infrared light source 118 may be modulated at a particularfrequency. Other types of modulation of infrared light source 118 arepossible.

HMD-tracking system 104 could be configured to provide a HMD positionand a HMD orientation to processor 112. This position and orientationdata may help determine a central axis to which a stare axis iscompared. For instance, the central axis may correspond to theorientation of HMD 100.

Gyroscope 120 could be a micro-electromechanical system (MEMS)gyroscope, a fiber optic gyroscope, or another type of gyroscope knownin the art. Gyroscope 120 may be configured to provide orientationinformation to processor 112. GPS unit 122 could be a receiver thatobtains clock and other signals from GPS satellites and may beconfigured to provide real-time location information to processor 112.HMD-tracking system 104 could further include an accelerometer 124configured to provide motion input data to processor 112.

Optical system 106 could include components configured to providevirtual images at a viewing location. The viewing location maycorrespond to the location of one or both eyes of a wearer of HMD 100.These optical system components 126, 128, and 130, introduced above, maybe optically and/or electrically-coupled to one another and may beconfigured to provide viewable images at a viewing location. Asmentioned above, one or two optical systems 106 could be provided in aHMD apparatus. In other words, the HMD wearer could view virtual imagesin one or both eyes, as provided by one or more optical systems 106.Also, as described above, optical system(s) 106 could include an opaquedisplay and/or a see-through display, which may allow a view of thereal-world environment while providing superimposed virtual images.

Various peripheral devices 108 may be included in HMD 100 and may serveto provide information to and from a wearer of HMD 100. In one example,HMD 100 may include a wireless communication interface 134 forwirelessly communicating with one or more devices directly or via acommunication network. For example, wireless communication interface 134could use 3G cellular communication, such as CDMA, EVDO, GSM/GPRS, or 4Gcellular communication, such as WiMAX or LTE. Alternatively, wirelesscommunication interface 134 could communicate with a wireless local areanetwork (WLAN), for example, using WiFi. In some embodiments, wirelesscommunication interface 134 could communicate directly with a device,for example, using an infrared link, Bluetooth, or ZigBee. Wirelesscommunication interface 134 could interact with devices that mayinclude, for example, components of HMD 100 and/or externally-locateddevices.

Although FIG. 1 shows various components of HMD 100 (i.e., wirelesscommunication interface 134, processor 112, memory 114, infrared camera116, display panel 126, GPS 122, and user interface 115) as beingintegrated into HMD 100, one or more of these components could bephysically separate from HMD 100. For example, infrared camera 116 couldbe mounted on the wearer separate from HMD 100. Thus, HMD 100 could bepart of a wearable computing device in the form of separate devices thatcan be worn on or carried by the wearer. The separate components thatmake up the wearable computing device could be communicatively coupledtogether in either a wired or wireless fashion.

FIG. 2A presents a front view of a head-mounted display (HMD) 200 in anexample embodiment that includes a head-mounted support 201. FIGS. 2Band 2C present the top and side views, respectively, of the HMD in FIG.2A. Although this example embodiment is provided in an eyeglassesformat, it will be understood that wearable systems and HMDs may takeother forms, such as hats, goggles, masks, headbands and helmets.Head-mounted support 201 includes a center frame support 206, lenselements 202 and 204, and extending side-arms 228 and 230. Center framesupport 206 and side-arms 228 and 230 are configured to securehead-mounted support 201 to the wearer's head via the wearer's nose andears, respectively. Center frame support 206 and extending side-arms 228and 230 may be formed of a solid structure of plastic or metal, or maybe formed of a hollow structure of similar material so as to allowwiring and component interconnects to be internally routed through thehead-mounted support 201. Alternatively or additionally, head-mountedsupport 201 may support external wiring. Lens elements 202 and 204 areat least partially transparent so as to allow the wearer to look throughthem. In particular, wearer's left eye 210 may look through left lens204 and the wearer's right eye 208 may look through right lens 202. Oneor more optical systems configured to display images to a HMD wearercould be incorporated into or associated with right lens 202 and leftlens 204 so that the HMD wearer could observe overlaid virtual images ona field of view of the real-world environment. Other ways of usingoptical systems for use as heads up displays (HUDs) or HMDs are known inthe art.

Although this example embodiment includes an optical system for each ofthe wearer's eyes, it is to be understood that an HMD might include anoptical system for only one of the wearer's eyes (either left eye 210 orright eye 208). In this example embodiment, right lens 202 and left lens204 may act as respective attachment locations for light sources 216 and218 as well as for cameras 224 and 226. Light sources 216 and 218 mayilluminate viewing location(s) 208 and 210, i.e. the wearer's eye(s),and may produce glint reflections from the eyes 208 and 210 that mayappear as bright glints 220 and 222, respectively, in the wavelengthrange of illumination. The number of light sources like light sources216 and 218 may vary. Cameras 224 and 226 may be infrared cameras, andmay image eyes 208 and 210, eye pupils 212 and 214, and glints 220 and222, respectively.

HMD 200 may include various elements such as a processor 232, a touchpad234, a microphone 236, and a button 238. Processor 232 may use datafrom, among other sources, various sensors and cameras to determine thevirtual image that should be displayed to the user. Further, lightsources 216 and 218 may represent one or more LEDs located at differentlocations in, and/or separate from, right lens 202 and left lens 204,respectively. The images of glints 220 and 222 may be transmitted toprocessor 232, which may accordingly determine a stare direction of awearer of the HMD from the relative and absolute locations of glints 220and 222. Processor 232 may also determine a stare point location, whichmay correspond to where the HMD wearer may be staring at on the displayarea of HMD 200. The stare point locations could be determined by, forinstance, the intersection of the physical or apparent display area andthe stare direction. These stare point locations (or more directly,stare locations) may be used by processor 232 to determine inputinstructions that could correspond to various functions, for instance,to increase the amount or type of information displayed once the userhas shown interest in a particular target or piece of information, whichis either displayed in an area of the display unit of HMD 200 or avisible through the see-through display unit.

Thus, light sources 216 and 218, cameras 224 and 226, and processor 232may function as a stare-acquisition system for determining staringdirections from images acquired by cameras 224 and 226 and fordetermining stare locations based on where the stare directions.Additionally, depending upon the particular embodiment, it may be thecase that a stare-acquisition system is made available to one eye of theHMD wearer and not to both eyes of the HMD wearer.

Those skilled in the art will understand that other user input devices,user output devices, wireless communication devices, sensors, andcameras may be reasonably included in such a wearable computing system.

3. A Method for Using Eye-Tracking Data to Produce Changes inInformation Density and Type.

An example method 300 is provided for producing changes in informationdensity and type about a target displayed on or visible through HMD 100and 200 based on eye-tracking data. The information changes may beperformed using the apparatus shown in FIGS. 1-2C and described above;however, other configurations could be used. FIG. 3 illustrates thesteps in example method 300, however, it is understood that in otherembodiments, the steps may appear in different order and steps may beadded or subtracted.

At step 302, computing system 101 is configured to image at least oneeye of the wearer in the manner discussed above in order to determine astare direction within a field of view of HMD 100, which includesdisplay unit 126 that may be a see-through unit. That is, to determinestare directions one or both eyes of the wearer of HMD 100 and 200 couldbe imaged. Specifically, infrared camera 116, 224 and 226 and one ormore infrared light sources 118, 216 and 218 could be used to acquireimages of the HMD wearer's eye(s). These images could be sent toprocessor 112. As discussed above, processor 112 could determine a staredirection from the glint locations on the HMD wearer's eye(s). From thedetermined stare direction, stare locations may be determined that couldcorrespond to locations where the HMD wearer may be looking at withinthe display area or at the real world. Based on the imaging of at leastone eye of the wearer, computing system 101 may also be configured todetermine a stare depth. The wearer's stare may have been directed at atarget that figures on a display presented on one of lenses 202 and 204.In this case, the depth is determined to be substantially equal to adistance separating the display from the wearer's eyes. Alternatively,the wearer's stare may have been directed at a target that is locatedwithin the real world seen through the see-through lenses 202 and 204,and the determined depth is thus greater than the distance separatingthe see-through lenses 202 and 204 from the wearer's eyes.

At step 304, computing system 101 is configured to identify the targetof interest and may be an environment of the target of interest based onthe stare direction and the stare depth. In case, the target of interestis part of the display, then the context is already known to processor112. Alternatively, in case the target is located in the real worldfacing/surrounding the wearer, then computing system 101 is configuredto identify the wearer's surroundings by extracting its image from theimaging of at least one eye of the wearer and from GPS data provided byGPS 122, and comparing it to environment data stored in a databaseassociated with memory 114 and/or in a database stored in a remoteserver (not shown) accessible via wireless communication interface 134.Alternatively, the remote server may be accessible via a wiredconnection.

At step 306, once the target has been identified, computing system 101is configured to determine a length of time during which the user hasbeen staring at the identified target, and to compare the determinedlength of time to a predetermined stare time threshold. Upon adetermination that the user's stare lasted more than the predeterminedstare time threshold, computing system 101 is configured to identifyinformation relevant to the target. Computing system 101 is thenconfigured to display the identified information over either a displayedimage containing the identified target or on a different area than thatof the displayed image, at step 308. In case the target is located inthe real world, the identified information may be displayed on displayunit 126 over a portion of the view of the real world.

FIGS. 4A-4C collectively depict an example of a progressive increase inthe amount of information displayed once the wearer has shown interestat a target through a stare, and keeps on staring after each additionalinformation display. The target may be a particular piece of informationor a physical object. In particular, FIGS. 4A-4C illustrate how thelonger the wearer's stare lasts the more information or detailscomputing system 101 identifies and presents on display unit 126. Manyother embodiments are possible and the described example embodiments arenot meant to limit the ways in which computing system 101 can identifytargets of interest and provide related information.

In FIG. 4A, an example embodiment of an HMD wearer in a stare-basedinteraction with a work of art is shown. The work of art may be apainting or photograph of a portion of the city of Chicago, Ill. Asillustrated, the HMD wearer is positioned in front of the city ofChicago painting 402, and looking at it through lens 204. While lookingat painting 402, the wearer's eyes are in a position that permits camera116 of eye tracking system 102 to track their movements and determinewhen a staring behavior occurs. As described above, a stare pointlocation may be determined from camera images of the HMD wearer'seye(s). Specifically, the stare point location may correspond to theintersection between the physical or apparent display area and the staredirection, which can be determined from the location of glints on theHMD wearer's eye(s).

Referring back to FIG. 4A, when the wearer directs his/her eye andstares at an area of interest 404 within painting 402, the eye movementinformation obtained by eye tracking system 102 is used to determinearea of interest 404, which includes the Willis Tower for example, andprovides an image of area of interest 404 to processor 112. If thewearer's stare is determined to last longer than a certain length oftime T1, the Willis Tower is identified and related information 406 isretrieved by processor 112 from memory 114 or from a database associatedwith a remote server accessible via wireless interconnection interface134, and projected/presented on lens 204. Alternatively, Willis Towerinformation may be presented on lens 202. This information may includehistorical data of the Willis Tower, such as its height, and itsprevious name, i.e., Sears Tower. Alternatively, the retrievedinformation may also include its inauguration date, and when it lost itstitle as the tallest building in the world.

Following the presentation of the retrieved information about the WillisTower, computing system 101 determines whether the wearer continuesstaring at the Willis Tower, at step 310. If the continued stare isdetermined to last longer than another length of time T2 subsequently tolength of time T1, computing system 101 is configured to identifyadditional information relevant to the Willis Tower. The additionalinformation may include more details about the Willis Tower, such as aportion of its architectural structure as depicted in an image 408 shownin FIG. 4B. Computing system 101 is then configured to display image 408on either lens 202 or 204 or on both, at step 312. In one embodiment,the stare times T1 and T2 may not be contiguous. That is, a second stareat the Willis Tower need not be contiguous to a first stare at theWillis Tower.

Still referring to FIG. 3, following the presentation of the additionalinformation about the Willis Tower, computing system 101 determineswhether the wearer still continues to stare at the Willis Tower, at step310. If the continued stare is determined to last longer than anotherlength of time T3 subsequently to length of time T2, computing system101 is configured to identify extra information relevant to the WillisTower. The extra information may include still more details about theWillis Tower, such as its bundled-tube structural design for example asdepicted in an image 410 shown in FIG. 4B. Computing system 101 is thenconfigured to display image 410 on either lens 202 or 204 or on both, atstep 312. As such, the longer the gaze at a particular target the moreinformation about the particular target are identified and presented tothe HMD wearer.

In one embodiment, following the presentation of the retrievedinformation about the Willis Tower, the wearer may continue staring atthe displayed retrieved information rather than redirecting his/herstare back at the Willis Tower or at another target. If the stare at thedisplayed information is determined to last longer than anotherpredetermined length of time, computing system 101 is configured toidentify additional information relevant to the Willis Tower that maybuild on or may be related to the displayed information in order toprovide additional details about a certain feature or features of thetarget.

In another embodiment, information that can be retrieved about a targetmay be arranged and stored as a list of information items. Each of theseinformation items may build on or may be related to one or more of theother information items. During operation, computing system 101 maycycle through the list of information items to display them in sequenceas the stare continues. As such, with each subsequent stare or as thesame staredown lengthens the last displayed information item is moved orshifted to the back of the list while the topmost information item ofthe list replaces the last displayed information item on either lens 202or 204 or on both. Thus, this combination of shifting and displaying ofinformation items may last as long as the wearer continues staring atthe target.

FIG. 5A is an example embodiment of an image of an email 502 displayedon one of lenses 202 and 204. As shown displayed on lens 204, email 502includes a “From” area, a “To” area, a “CC” area, a “Subject” area, anda “content” area. FIG. 5A illustrates a stare point location 504, whichmay be determined by computing system 101 to be near the Subject area.Accordingly, upon detection of the staring at the Subject area for atime at least equal to a predetermined amount of time T4, for instance500 milliseconds, computing system 101 is configured to retrieve thetime 506 at which email 502 was send by a sender, and to display it overemail 502 or over any other area of lens 204, as shown in FIG. 5A.

Now referring to FIG. 5B, following the presentation of the retrievedinformation about email sending time 506, computing system 101determines whether the HMD wearer continues staring at email 502. If thecontinued stare is determined to last longer than another length of timeT5 subsequently to length of time T4, computing system 101 is configuredto identify additional information relevant to email 502. For example,the additional information may include more details about email 502,such as a portion of its email header that discloses who the real senderof email 502 was. As such, computing system 101 is configured toretrieve the header information, and presents on lens 204 an InternetProtocol (IP) address 508 of the sender.

In another example embodiment, the HMD wearer may be strolling through azoological park and posing in front different animal holding areas orcages. When the HMD wearer stops to look at a particular animal, thewearer's eyes may be imaged by infrared cameras 224 and 226 and found bycomputing system 101 to be staring at the particular animal. In thisscenario, computing system 101 is configured to capture an image of theparticular animal. Based on the captured image and on a staring durationthat lasts at least as long as a predetermined amount of time T6,computing system 101 may retrieve information about the particularanimal.

Following the presentation of the retrieved information about theparticular animal, computing system 101 determines whether the HMDwearer continues staring the particular animal. If the continued stareis determined to last longer than another length of time T7 subsequentlyto length of time T6, computing system 101 may be configured to identifyadditional information relevant to the particular animal, such as forexample its muscular system. Following the presentation of the muscularsystem of the particular animal, computing system 101 determines whetherthe HMD wearer continues staring the particular animal. If the continuedstare is determined to last longer than another length of time T8subsequently to length of time T7, computing system 101 may beconfigured to identify additional information relevant to the particularanimal, such as for example its skeletal system. As such, the longer thegaze the more information at the particular animal is identified andpresented to the HMD wearer.

4. Non-Transitory Computer Readable Medium

In some embodiments, the disclosed methods may be implemented ascomputer program instructions encoded on a computer-readable storagemedia in a machine-readable format. FIG. 6 is a schematic illustrating aconceptual partial view of an example computer program product 600 thatincludes a computer program for executing a computer process on acomputing device, arranged according to at least some embodimentspresented herein. In one embodiment, the example computer programproduct 600 is provided using a signal bearing medium 601. The signalbearing medium 601 may include one or more programming instructions 602that, when executed by one or more processors may provide functionalityor portions of the functionality described above with respect to FIG.1-5B. Thus, for example, referring to the embodiment shown in FIG. 3,one or more features of blocks 302, 304, 306, 308, 310, and/or and 312may be undertaken by one or more instructions associated with the signalbearing medium 601.

In some examples, the signal bearing medium 601 may encompass anon-transitory computer-readable medium 603, such as, but not limitedto, a hard disk drive, a Compact Disc (CD), a Digital Video Disk (DVD),a digital tape, memory, etc. In some implementations, the signal bearingmedium 801 may encompass a computer recordable medium 604, such as, butnot limited to, memory, read/write (R/W) CDs, R/W DVDs, etc. In someimplementations, the signal bearing medium 801 may encompass acommunications medium 605, such as, but not limited to, a digital and/oran analog communication medium (e.g., a fiber optic cable, a waveguide,a wired communications link, a wireless communication link, etc.). Thus,for example, the signal bearing medium 601 may be conveyed by a wirelessform of the communications medium 605 (e.g., a wireless communicationsmedium conforming to the IEEE 802.11 standard or other transmissionprotocol).

The one or more programming instructions 602 may be, for example,computer executable and/or logic implemented instructions. In someexamples, a computing device such as the computing device 102 of FIG. 1may be configured to provide various operations, functions, or actionsin response to the programming instructions 602 conveyed to thecomputing device 102 by one or more of the computer readable medium 603,the computer recordable medium 604, and/or the communications medium605.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopeand spirit being indicated by the following claims, along with the fullscope of equivalents to which such claims are entitled. It is also to beunderstood that the terminology used herein is for the purpose ofdescribing particular embodiments only, and is not intended to belimiting.

The above detailed description describes various features and functionsof the disclosed systems, devices, and methods with reference to theaccompanying figures. While various aspects and embodiments have beendisclosed herein, other aspects and embodiments will be apparent tothose skilled in the art. The various aspects and embodiments disclosedherein are for purposes of illustration and are not intended to belimiting, with the true scope and spirit being indicated by thefollowing claims.

What is claimed is:
 1. A computer-implemented method comprising:detecting, at a wearable computing device, a first direction of a firststare, wherein the wearable computing device includes a head-mountabledisplay unit; identifying a target based on the detected firstdirection; based on a determination that a first time duration of thefirst stare is greater than or equal to a first predetermined timethreshold, identifying information relevant to the target and displayingthe identified information on the head-mountable display unit;subsequent to displaying the identified information, detecting a secondstare that is directed at the target or at the displayed information,wherein the first stare and the second stare are contiguous in time; andbased on a determination that a second time duration of the second stareis greater than or equal to a second predetermined time threshold,identifying additional information relevant to the target, anddisplaying the additional information on the head-mountable displayunit.
 2. The method of claim 1, wherein detecting the first direction ofthe first stare comprises: obtaining at least one image of each eye of awearer of the wearable computing device; and determining the first staredirection from the at least one image of each eye.
 3. The method ofclaim 2, wherein obtaining the at least one image of each eye of thewearer of the wearable computing device comprises: illuminating each eyewith an infrared light source; and imaging each eye with a camera. 4.The method of claim 1, wherein identifying the target comprises:capturing an image of the target; and performing a target recognitionbased on the captured image.
 5. The method of claim 1, wherein the headmountable display unit comprises a see-through display unit.
 6. Themethod of claim 1, wherein the wearable computing device is ahead-mounted device.
 7. The method of claim 1 further comprising:progressively displaying more additional information about the targetwhile the stare continues.
 8. A computer-implemented method comprising:detecting, at a wearable computing device, a direction of a stare,wherein the wearable computing device includes a head-mountable displayunit; identifying a target based on the detected direction; based on adetermination that a first time duration of the stare is greater than orequal to a first predetermined time threshold, identifying informationrelevant to the target and displaying the identified information on thedisplay unit; subsequent to displaying the identified information,determining that a second time duration of the stare is greater than orequal to a second predetermined time threshold; based on a determinationthat the second time duration of the stare is greater than or equal tothe second predetermined time threshold, identifying additionalinformation relevant to the target, and displaying the additionalinformation on the display unit.
 9. The method of claim 8, whereindetecting the direction of the stare comprises: obtaining at least oneimage of each eye of a wearer of the wearable computing device; anddetermining the stare direction from the at least one image of each eye.10. The method of claim 9, wherein obtaining the at least one image ofeach eye of the wearer of the wearable computing device comprises:illuminating each eye with an infrared light source; and imaging eacheye with a camera.
 11. The method of claim 8, wherein identifying thetarget comprises: capturing an image of the target; and performing atarget recognition based on the captured image.
 12. The method of claim8, wherein the head-mountable display unit comprises a see-throughdisplay unit.
 13. The method of claim 8 further comprising:progressively displaying more additional information about the targetwhile the stare continues.
 14. The method of claim 8, wherein thewearable computing device is a head-mounted device.
 15. Acomputer-implemented method comprising: detecting, at a wearablecomputing device, a direction of a stare, wherein the wearable computingdevice includes a head-mountable display unit; identifying a targetbased on the detected direction; based on a determination that a firsttime duration of the stare is greater than or equal to a firstpredetermined time threshold, identifying information relevant to thetarget and displaying the identified information on the head-mountabledisplay unit; and subsequent to displaying the identified information,progressively displaying additional information about the target whilethe stare continues, comprising: determining that a second time durationof the stare is greater than or equal to a second predetermined timethreshold; and based on a determination that the second time duration ofthe stare is greater than or equal to the second predetermined timethreshold, identifying additional information relevant to the target,and displaying the additional information on the head-mountable displayunit.
 16. The method of claim 15, wherein detecting the direction of thestare comprises: obtaining at least one image of each eye of a wearer ofthe wearable computing device; and determining the stare direction fromthe at least one image of each eye.
 17. The method of claim 16, whereinobtaining the at least one image of each eye of the wearer of thewearable computing device comprises: illuminating each eye with aninfrared light source; and imaging each eye with a camera.
 18. Themethod of claim 15, wherein identifying the target comprises: capturingan image of the target; and performing a target recognition based on thecaptured image.
 19. The method of claim 15, wherein the head-mountabledisplay unit comprises a see-through display unit.
 20. The method ofclaim 15, wherein the wearable computing device is a head-mounteddevice.